-
Notifications
You must be signed in to change notification settings - Fork 31
WIP: ci/e2e-tests: Publish HTML log and report on GitHub Pages #1236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
adombeck
wants to merge
47
commits into
main
Choose a base branch
from
e2e-tests-gh-pages
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes our life easier when trying to upload the test outputs as artifacts on CI.
The syntax used in this execution was flawed.
This is a QoL change to help us fine tune the artifacts in CI.
We were using only the base name to create the image file, which could cause conflicts if we ever try to setup both releases.
The 'await' command on virsh is not available on Noble, which is our runner in github, so we need to use another way to check for the domain status.
Update the commands to also run `sudo apt-get update` and add some dependencies that were missing from the lists
This workflow downloads an Ubuntu image, sets up some of the aplications and configurations that we will need in the VM in order to be able to run the E2E tests and then caches it for later use to save us some time when running the tests. Co-authored-by: Adrian Dombeck <[email protected]>
fix output artifact naming ci/e2e-tests: Use env vars instead of config file The config file is useful when running the tests locally, but it only sets environment variables, so in the CI it's simpler to just set the environment variables directly. ci/e2e-tests: Fix indentation
This caused the provisioning to fail
We were padding the generated code with trailing 0s, but we should have been padding with leading 0s. This commit fixes that.
The code is generated according to the current time and is valid for 30 seconds. This means that if we generate the code just before the time window changes, it might be invalid by the time we use it. To avoid this, we make sure the time is safely within a new window before generating the code.
We see permission denied errors when trying to start the VM both in CI and locally because AppArmor denies access to the qcow2 image (curiously, starting the VM by restoring a running snapshot works). We should investigate the issue and report it upstream. In the meantime, let's work around it by disabling AppArmor confinement for the e2e-test VM. That should be fine since we're not doing anything in the VM which would make them more vulnerable than the host.
We recently changed the error message that is shown when the broker fails to reach the provider, so we need to update this check on the tests as well.
Sometimes machinectl login hangs and it causes the tests to fail. Let's try reopening it a couple of times (in case of failure) to see if we are able to get more consistent behavior
systemd only supports sending the journal via VSOCK since v256. In the GitHub CI the newest Ubuntu release we can use on the runners (both GitHub-hosted or self-hosted) is 24.04, which has systemd 255.
Running the tests is quite resource intensive, so let's switch to using our self-hosted runners to see if we can get some improvements in performance and stability.
It doesn't only "Build and Cache Ubuntu VM", it runs the e2e-tests and does everything required to do so.
On GitHub Runners, the VM can't connect to the host via TCP. This commit instead makes the host connect to the VM to retrieve the journal.
On the self-hosted runners, the tests fail because the SSH connection to the VM fails. Also, downloading the VM image from the artifacts takes 12 minutes instead of 20 seconds. This reverts commit 4201248.
The tests are also useful when targeting other branches than main.
There are not many files in the repo which don't affect the e2e-tests (once we run them with authd and broker artifacts built from the branch) but it's still worthwhile to add include/exclude paths because the tests are very resource intensive.
To avoid duplication between the msentraid and google e2e-tests.
So that we automatically re-provision the VM image if anything changed in the relevant files.
It's too much output
It's not only testing authd-oidc-brokers but also authd.
We ignored errors during cloud-init and happily continued provisioning a broken VM.
Should save some time and bandwidth.
The file is already executable
61c7d29 to
9f93b5a
Compare
1caa564 to
b8d8043
Compare
b8d8043 to
5feb701
Compare
5feb701 to
a0eeaf2
Compare
a0eeaf2 to
dc18913
Compare
9f93b5a to
0e7717f
Compare
fba8d56 to
b4bf91b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.